Multi-path routing using intra-flow splitting

ABSTRACT

Multi-path routing techniques using intra-flow splitting are disclosed. For example, a technique for processing traffic flows at a node in a network comprises the following steps/operations. At least one traffic flow is obtained. The at least one traffic flow comprises multiple packets or bytes. The at least one flow is split into at least two sub-flows, wherein each of the at least two sub-flows comprises a portion of the multiple packets or bytes. The packets or bytes of the at least two sub-flows are respectively routed on at least two paths in the network.

FIELD OF THE INVENTION

The present invention relates generally to the field of datacommunication networks and, more particularly, to multi-path routingtechniques using intra-flow splitting.

BACKGROUND OF THE INVENTION

The past decade has seen a tremendous growth in the amount of datatraffic carried over wide area networks. This trend is expected tocontinue with the growing popularity of the Internet and the emergenceof new applications such as Voice-over-Internet Protocol (VoIP).

In addition to the rising traffic volumes, there has also been anevolution in the underlying networks. From the optical transport layerand up, network topologies have become more mesh-like, allowing multiplediverse paths between source and destination nodes. This diversity isessential in providing resiliency for critical demands via backup paths.A high degree of connectedness in a network also allows sharing oftraffic load across various links and demands, and hence better networkutilization. This is important because long-haul bandwidth continues tobe expensive due to the high costs of Wavelength Division Multiplexing(WDM) transport systems and high speed router ports.

Load-sharing can be achieved in two complementary ways. One way isthrough congestion-aware routing algorithms to route the demands suchas, for example, Open Shortest Path First (OSPF) techniques. Another wayis by routing the packets of the same demand over multiple paths alongthe way. The latter, called multi-path routing, provides fast resiliencyas well as a finer degree of load sharing in the network. In fact, OSPFand other approaches have multi-path extensions, such as equal-costmultipath (ECMP) and optimized multipath (OMP), where routers distributethe incoming load on an interface over all available shortest paths. Itis to be understood that “shortest paths” generally refers to thecheapest paths under the cost metric chosen by the OSPF algorithm.

In multi-path routing, packets can be distributed using either around-robin mechanism or a hash function on the flow identifiers. Thehash-based approach routes all packets of a flow over the same path andmay lead to load imbalances due to variations in flow rates. While theround-robin scheme will lead to better load sharing, since the packetsof the same flow may be sent over different links, they can arriveout-of-order at the destination. If not resequenced, out-of-orderarrival of packets leads to increased dropping of packets by the higherlayer protocols (e.g., Transmission Control Protocol), as well as jitterin delay. Resequencing at high traffic rates, on the other hand,requires expensive processors and large memories. As a result, theround-robin mechanism has been mostly unused in practice.

Admission control and capacity planning in a network require an accurateknowledge of the bandwidth needed on each link to carry the giventraffic load. However, it is difficult to exactly compute the bandwidthneeds of variable bit-rate (VBR) traffic, such as most of the datatraffic. This is typically handled in practice via the concept of“effective bandwidth,” which is an estimate of the bandwidth needed tosatisfy a quality-of-service (QoS) requirement such as, for example, adrop rate, a maximum queuing delay, etc. Effective bandwidth depends onthe traffic characteristics, i.e., the average rate and the variability,as well as the strictness of the QoS requirement.

SUMMARY OF THE INVENTION

Principles of the present invention provide multi-path routingtechniques using intra-flow splitting.

For example, in one aspect of the invention, a technique for processingtraffic flows at a node in a network comprises the followingsteps/operations. At least one traffic flow is obtained. The at leastone traffic flow comprises multiple packets or bytes. The at least oneflow is split into at least two sub-flows, wherein each of the at leasttwo sub-flows comprises a portion of the multiple packets or bytes. Thepackets or bytes of the at least two sub-flows are respectively routedon at least two paths in the network.

Further, when at least a second traffic flow is obtained by the node,the splitting step/operation may further comprise splitting each of thetraffic flows into at least two sub-flows. The routing step/operationmay then further comprise combining ones of the sub-flows of at least aportion of the at least two traffic flows and routing the combinedpackets or bytes on one of the paths, and combining others of thesub-flows of at least a portion of the at least two traffic flows androuting the combined packets or bytes on another of the paths. Thesplitting step/operation may further comprise splitting the at least onetraffic flow based on a split ratio vector.

Still further, splitting of the flows, combining sub-flows, and routingthe packets or bytes of the combined sub-flows on at least two paths inthe network may reduce a variance, loss probability, and a bandwidthrequirement associated with the traffic flows. Also, the at least onetraffic flow may be long range dependent or short range dependent.

These and other objects, features and advantages of the presentinvention will become apparent from the following detailed descriptionof illustrative embodiments thereof, which is to be read in connectionwith the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a network employing flow-based splitting;

FIG. 1B illustrates a network employing intra-flow splitting, accordingto one embodiment of the invention;

FIG. 2 illustrates an intra-flow splitting methodology, according to oneembodiment of the invention;

FIG. 3 illustrates a two-node network for use in describing one or moreembodiments of the invention; and

FIG. 4 illustrates an implementation of a node, according to oneembodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

The following description will illustrate the invention in the contextof an exemplary packet-switched mesh network. It should be understood,however, that the invention is not necessarily limited to use with anyparticular type of network. The invention is instead more generallyapplicable to any network in which it is desirable to provide improvedmulti-path routing.

Further, it is to be understood that the phrase “traffic flow” (or,simply, “flow”) generally refers to a group (e.g., two or more) ofpackets that are to be routed in the network through the same sourcenode/port and destination node/port pairing. A “path” in the networkgenerally refers to a set of two or more nodes and one or more linksbetween the nodes. “Nodes” generally refer to elements in the network atleast capable of transferring packets. “Links” generally refer toconnections between the nodes.

As will be illustratively explained in the sections of the detaileddescription below, principles of the invention provide multi-pathrouting techniques that perform intra-flow splitting. Advantageously, asignificant variance reduction is realizable due to such splitting,which directly leads to much lower effective bandwidth needs as comparedwith flow-based splitting schemes. Bandwidth savings is illustrativelyquantified for various traffic models.

1. Multi-Path Routing and Traffic Modeling

In this section, different classes of multi-path routing are introduced,as well as various traffic modeling concepts used in an illustrativebandwidth analysis. Also, an illustrative intra-flow splittingmethodology is described.

1.1 Network Model

In this detailed description, a packet-switched mesh network isillustratively assumed where traffic flows are generated by sourceshaving specific statistical characteristics. A traffic flow isidentified by a vector made up of a source node, a source port, adestination node, a destination port. Each node routes the packets basedon a flow identifier.

Each link has certain finite capacity. When packets arrive at a rategreater than the link capacity, they are queued in a buffer. In theillustrative model, it is assumed that there is a finite buffer of sizeB bytes per link. Nodes drop packets when the queue is full. It is notcritical to the analysis which packets are chosen for dropping.

When there are multiple paths from a source to destination, packets arerouted across these multiple paths. The splitting can happen at any ofthe nodes along the way. For purposes of comparison, we consider twoforms of splitting: (i) flow-based splitting; and (ii) intra-flowsplitting in accordance with principles of the invention.

In flow-based splitting, packets of a flow take the same path. Note thattraffic between the same source and destination can consist of multipleflows which may take different paths. An example of flow based splittingis hash-based ECMP, where the nodes compute a hash function on theflow-identifiers to determine the outgoing link.

In intra-flow splitting, packets of the same flow may be sent overdifferent paths. Here, each path will carry a fraction of the trafficflow. This fraction may be decided based on link capacities and/or thecurrent load.

For intra-flow splitting, a split ratio vector of a flow is defined asthe proportion of the flow routed in each path. Let f be a traffic flowand K be its number of paths (not necessarily disjoint) from the sourceto destination. Let p_(i) fraction of the flow be routed through thei^(th) path (Σp_(i)=1). This fraction can be in terms of bytes orpackets. The fraction in terms of packets is used unless notedotherwise. The split ratio vector of f is called (p₁, p₂, . . . ,p_(k)). If p_(i)=1/K for i=1 . . . K, then intra-flow splitting for theflow is round-robin. In that case we define K as the split factor.

It is to be understood that intra-flow splitting of the invention isdifferent than round-robin ECMP in that intra-flow splitting of theinvention is deterministic or intentional in nature. That is, while inround-robin ECMP, packets of the same flow may happen to be routed ondifferent paths, this is not done intentionally or evendeterministically calculated. That is, the breaking up of packets of aflow and routing them on different paths is merely a result of the factthat part of the packets of a flow may be received at a different timethan other packets of the same flow. In such a case, the packetsreceived first in time are assigned to the path that is currentlyavailable in the round-robin ECMP scheme, while the packets receivedlater in time are assigned to the path that is available at that time.Under the round-robin ECMP scheme, this could be the same path or itcould be a different path. Thus, there is no notion in round-robin ECMPof purposefully splitting flows into sub-flows.

FIGS. 1A and 1B illustrate the two forms of splitting. Moreparticularly, FIG. 1A illustrates flow-based splitting and FIG. 1Billustrates intra-flow splitting. In the example shown, there are twoflows X₁ and X₂ destined to the same destination R₆. There are two pathsfrom R₁ to R₆.

In a flow based splitting scheme, flow X₁ takes the path R₁-R₂-R₃ -R₆and the flow X₂ takes the path R₁-R₄-R₅-R₆.

In intra-flow splitting, flow X₁ is split at R₁ with the split ratiovector (q₁, 1-q₁). Then, q₁ fraction of the flow takes the pathR₁-R₂-R₃-R₆ and the remaining takes the path R₁-R₄-R₅-R₆. Similarly forthe flow X₂, the traffic is split with the split ratio vector (q₂,1-q₂).

More particularly, FIG. 2 illustrates an intra-flow splittingmethodology according to the invention. As shown, at the source node, aflow is received (step 210). The packets of the flow are split (step220), as explained above based on the split ratio vector, into two ormore sub-flows. Then, the two or more sub-flows are routed on two ormore selected paths (step 230) to the destination node. When there aremore than two flows received at the source node, as shown in FIG. 1B,the steps of FIG. 2 can be performed on each flow, and sub-flows fromdifferent flows can be multiplexed or combined together and routed.

Note that the intra-flow splitting techniques of the invention not onlyallow splitting on exactly equal cost paths, they also allow splittingeven if the paths are of nearly the same cost. This may lead to somelonger paths, but bandwidth savings is still realized.

1.2 Traffic Model

1.2.1 Traffic Superposition

An important issue in traffic modeling is how to characterize theaggregate traffic going on the core network. As a background anillustrative model below, consider the Internet where packets aregenerated by the activities of individual users. Suppose that the userpopulation is mostly homogeneous and each user acts independent of theother. The superposition model given below is a natural way of modelingthe aggregate traffic in such common scenarios.

We assume that the traffic flows are stationary in short durations. LetB_(i)(t) be the base traffic process between a pair i of users, whichgives the number of bytes transferred in a time interval t. We assumethat B_(i)(t)s are independent and identically distributed. Wecharacterize traffic on the network as a superposition of trafficoriginating from multiple sources. Let Y(t) be the traffic process forany flow from a source to a destination, then we model Y(t) assuperposition of the m independent and identically distributed basetraffic processes, i.e.:Y(t)=B ₁(t)+ . . . +B _(m)(t),  (1)

We assume that for any two distinct flows f₁ and f₂, the traffic processY₁(t) and Y₂(t) are independent.

For a flow Y(t), let the mean traffic rate be μ_(y), and let theautocovariance function be σ_(y)r(t), where r(t) is the autocorrelationfunction. Note that the values of μ_(y) and σ_(y) change with the flowbut the autocorrelation function does not, because the base processesare independent and identically distributed. However, under thesuperposition model (Equation 1), it can be shown that μ_(y) is relatedto σ_(y) by:σ_(y) ²=αμ_(y,)  (2)

for some constant α>0. In addition, this variance-mean relationship alsoholds under the further superposition of these aggregated flows. For aGaussian traffic model completely characterized by the mean andcovariance function, the flow model of Y(t) in Equation 1 can berewritten more succinctly as follows:Y(t)=μ_(y)+σ_(y) Z(t), σ_(y) ²=αμ_(y),  (3)

where Z(t) is a stationary Gaussian process with variance 1.

1.2.2 Short-Range and Long-Range Dependent Model

Let X(t) be an ergodic and stationary traffic process, and let X[0, t]be the cumulative traffic arriving in the interval [0,t]. Define thevariance time function of X(t) as V(t)=Var(X[0, t]). Define the index ofdispersion of X(t) as:v= _(t→∞) ^(lim) V(t)/t,  (4)

then X(t) is said to be Short Range Dependent (SRD) if v is finite orLong Range Dependent (LRD) if otherwise. It can be shown that for a LRDprocess, the variance time function has the form:v(t)→L(t)t^(2H) αs t→∞,

where L(t) is a slowly varying function and parameter H is referred toas the Hurst parameter in the literature. Notice that LRD occurs for1/2<H<1, and higher the value of H, the stronger the long-rangedependence.

The discovery of long-range dependence in Ethernet traffic is consideredto be one of the most significant in the area of traffic modeling.Further studies confirm that metropolitan area network (MAN) traffic,wide area network (WAN) traffic, and variable bit rate (VBR) trafficalso exhibit LRD.

1.2.3 Fractional Brownian Motion Model

A popular class of LRD traffic model is the Fractional Brownian Motion(FBM). It has been shown that if traffic is the sum of on-off processeswith heavy-tailed on/off period, then the traffic is LRD and that undercertain limit conditions, the resulting traffic approaches the FBMprocess as the number of aggregation grows large.

Let X(t) be a traffic source generated using:X[0, t]=μt+Z(t),  (5)

where X[0,t] is the total traffic arrived in the interval [0, t], andZ(t) is Gaussian process with zero mean. Then, X(t) is called FractionalBrownian Motion (FBM) with Hurst parameter Hε(0, 1) if VarZ(t)=σ²t^(2H). Notice that when 1/2<H<1, X(t) is LRD and when H=1/2,X(t) is reduced to a Brownian motion that is short-range dependent.

It is to be appreciated that during parts of the illustrative analysisto follow, we model data traffic as composed of a superposition ofmultiple long-range dependent flows, each of which is best modeled byFBM.

2. General Analysis

In this section, we present an intuitive argument for the bandwidthsavings from intra-flow splitting. The detailed analysis for varioustraffic models is given in Section 3 below.

2.1 Traffic Characteristics After Intra-Flow Splitting

Consider a source and destination node pair with K paths, where K>1. LetY(t) be a flow between the source-destination pair, then usingintra-flow splitting principles, with a splitting ratio vector (p₁, . .. , p_(k)), results in traffic p_(k)Y(t) on path k from flow Y(t).

For an existing routing scheme, let X_(k)(t) denote the cumulative flowon path k, k=1, . . . K, between the source-destination pair. It is easyto see that the total flow between the source and destination pair is

$\sum\limits_{i = 1}^{K}{{X_{i}(t)}.}$If we apply intra-flow splitting according to the invention, where allflows between the source-destination pair has the same splitting ratiovector (p₁, . . . , p_(k)), then the resulting traffic on path k is:

$\begin{matrix}{{{{\overset{\sim}{X}}_{k}(t)} = {p_{k}{\sum\limits_{i = 1}^{K}{X_{i}(t)}}}},\mspace{14mu}{K = 1},\ldots\mspace{11mu},K} & (6)\end{matrix}$that is, a proportion p_(k) of the multiplexed traffic between thesource-destination node pair. Below in section 3, we will show that itis in fact this multiplexing that results in the bandwidth savings seenin the intra-flow splitting approach, and we will quantify it using thenotion of effective bandwidth for Gaussian processes. In the following,we give an illustrative example of the bandwidth savings using thetwo-node network in FIG. 3.2.2 Bandwidth Savings

Consider the two-node network in FIG. 3. Suppose there are two parallellinks between a source node S and a destination node D, with each linkcarry traffic X_(k)(t), k=1,2 respectively. Suppose X₁(t), X₂(t) areindependent and identically distributed Gaussian processes, we canrewrite X_(k)(t), k=1, 2 as:X ₁(t)=μ+σZ₁(t), X ₂(t)=μ+σZ ₂(t)

where μ,σ² is the mean and variance, and Z₁(t), Z₂(t) are independentlydistributed as some process Z(t). Under the intra-flow splittingapproach where all flows have the same splitting ratio vector (1/2,1/2)on the two links, the resulting traffic on both links (by Equation 6)is:

${{{\overset{\sim}{X}}_{1}(t)} = {{{\overset{\sim}{X}}_{2}(t)} = {\frac{1}{2}\left( {{X_{1}(t)} + {X_{2}(t)}} \right)}}},$

which can be represented as:

${{\overset{\sim}{X}}_{1}(t)} = {{\mu + {\frac{\sigma}{2}\left( {{Z_{1}(t)} + {Z_{2}(t)}} \right)}} = {\mu + {\frac{\sigma}{\sqrt{2}}{{Z(t)}.}}}}$

Therefore under intra-flow splitting, the traffic on each link has thesame mean rate as in the original scheme but a reduced variance. It isclear from queuing theory that the resulting traffic {tilde over(X)}_(k)(t) requires less bandwidth than X_(k)(t) for the same QoSrequirement, and thus bandwidth savings is realized for the intra-flowsplitting.

In the above, we have given a very simple example explaining intuitionbehind the bandwidth savings of intra-flow splitting. The idea is thatintra-flow splitting results in traffic that has a higher degree ofmultiplexing on network links. If the intra-flow splitting approach isdefined such that the mean traffic rate on the link remains the same,then the smoothing effect introduced by increased multiplexing leads tobandwidth savings.

However, the above explanation dealt with a simple network and trafficdistribution, and does not actually quantify the savings. The nextsection analyzes the behavior more completely.

3. Analytical Results

In this section, we illustrate performance of the intra-flow splittingapproach of the invention. We first demonstrate that for a simpletwo-node network, intra-flow splitting gives the optimal queueingperformance, and hence realizes bandwidth savings over other routingschemes. Next, we compare the performance of intra-flow splitting withflow-based splitting for a general network and show that intra-flowsplitting performs better under certain traffic assumptions. Andfinally, we give specific results of the performance gain in terms ofloss probability and bandwidth savings for Gaussian traffic models andin particular the Fractional Brownian Motion traffic model. We assumethat the split ratio vectors for all the flows between a same source anddestination pair are the same.

3.1 Optimality for a Two-Node Network

It can be shown that there exists an intra-flow splitting that isoptimal in terms of queueing performance and bandwidth savings for a twonode network with parallel links. In particular, it can be shown thatthe optimal intra-flow splitting has the same split ratio vector for allthe flows and the fraction of flow on a link is proportional to the linkbandwidth.

Consider a simple two-node network with K links, e.g., networkillustrated in FIG. 3. Let link k, k=1, . . . , K have a bandwidth Ckand a buffer size B_(k) that is proportional to the bandwidth, i.e.B_(k)=αC_(k). It can be shown that the intra-flow splitting approachwith a splitting ratio vector that is proportional to the linkbandwidths has an optimal queuing performance for a two-node network,i.e., the steady state queue length and loss probability are thesmallest.

We show this using a discrete time traffic model, which can be used asan approximation for a continuous time model if the time unit goes to 0.Without loss of generality, in the following, we assume that the timeunit is 1.

For link k, 1≦k≦K, let X_(k)(t) be the total workload (bytes) arrivingduring the t-th time unit for an arbitrary routing scheme R, and letQ_(k)(t), k=1, . . . , K be the unfinished work at time t. Suppose thatthe queues for all links are empty at time t=0. It is well known that:Q _(k)(t+1)=min((Q _(k)(t)+X _(k)(_(t))−C _(k))⁺ , B _(k)),

where A⁺=max(0, A). For link k, if the mean traffic arrival rateE{X_(k)(0)} is less than the bandwidth C_(k), then Q_(k)(t) converges tothe steady state distribution of unfinished workload, denoted byQ_(k)(∞).

Let:

${{X(t)} = {\sum\limits_{k = 1}^{K}{X_{k}(t)}}},{C = {\sum\limits_{k = 1}^{K}C_{k}}},{B = {\sum\limits_{k = 1}^{K}{B_{k}.}}}$

For an intra-flow splitting approach with the same split ratio vector(p₁, . . . ,l p_(k)) for all flows, the resulting traffic on link k inthe interval t is p_(k)X(t). Let {tilde over (Q)}_(k)(t) be theunfinished workload of the approach at time t on link k. If the splitratio vector is such that p_(k) is proportional to-the link bandwidthC_(k), it is easy to see that {tilde over (Q)}_(k)(t)=p_(k){tilde over(Q)}(t), where {tilde over (Q)}(t) is the unfinished workload at time ton a link with bandwidth C and buffer size B carrying the total trafficX(t). Notice that:{tilde over (Q)}(t+1)=min(({tilde over (Q)}(t)+X(t)−C)⁺ , B),

and the fact that

$\left( {\sum\limits_{k = 1}^{K}y_{k}} \right)^{+} \leq {\sum\limits_{k = 1}^{K}{y_{k}^{+}.}}$using a recursive argument, it is shown that:

${\sum\limits_{k = 1}^{K}{{\overset{\sim}{Q}}_{k}(t)}} = {{\overset{\sim}{Q}(t)} \leq {\sum\limits_{k = 1}^{K}{{Q(t)}\mspace{14mu}{for}\mspace{14mu}{any}\mspace{14mu} t}} \geq 0.}$

Hence, for the steady state workload

${{\sum\limits_{k = 1}^{K}{{\overset{\sim}{Q}}_{k}(\infty)}} = {{\overset{\sim}{Q}(\infty)} \leq {\sum\limits_{k = 1}^{K}{Q_{k}(\infty)}}}},$which shows that the intra-flow splitting approach has the optimal queuelength.

To show that the intra-flow splitting approach has an optimal lossprobability, first notice that the amount of bytes dropped by therouting scheme R on link k during t^(th) time interval is(Q_(k)(t)+X_(k)(t)−C_(k)−B_(k))⁺. Let the steady state lossprobabilities for the routing scheme R and intra-flow splitting approachbe d and {tilde over (d)}, respectively. Then:

$\begin{matrix}{d = {\frac{1}{E\left\lbrack {X(0)} \right\rbrack}{\lim\limits_{n\rightarrow\infty}{\sum\limits_{k = 1}^{K}{E\left( \left( {{Q_{k}(n)} + {X_{k}(n)} - C_{k} - B_{k}} \right)^{+} \right)}}}}} \\{\geq {\frac{1}{E\left\{ {X(0\rbrack} \right.}{\lim\limits_{n\rightarrow\infty}{E\left( {{\sum\limits_{k = 1}^{K}{Q_{k}(n)}} + {X(n)} - C - B} \right)}^{+}}}} \\{{\geq {\frac{1}{E\left\lbrack {X(0)} \right\rbrack}\left( {{\overset{\sim}{Q}(n)} + {X(n)} - C - B} \right)^{+}}} = \overset{\sim}{d}}\end{matrix}$

Accordingly, the optimal queueing performance of the intra-flowsplitting approach can be translated to its optimal bandwidth savings.That is, for a two-node network, an intra-flow splitting approachrequires the least amount of total bandwidth to satisfy a QoS criteriaof a maximum queueing delay and a loss probability.

It is to be appreciated that the optimal performance of the intra-flowsplitting approach is the result of the multiplexing gain becauseintra-flow splitting essentially uses a single link to carry themultiplexed traffic by combining bandwidths and buffer sizes of themultiple links between two nodes.

3.2 Performance Improvements in a General Network

In this section, we illustrate the performance gain of intra-flowsplitting over flow-based splitting for a general network. We show thatunder the traffic assumption in section 1.2 (i.e., each flow is theresult of superposition of independent and identically distributed basetraffic flows), the intra-flow splitting approach performs better overflow-based splitting. We first demonstrate this for Gaussian trafficmodels and then generalize it to arbitrary traffic models.

For a general network, let R be a flow-based splitting scheme for agiven traffic demand. For any pair of a source node S and a destinationnode D, assume that there are K paths (distinct paths may share commonlinks), and let X_(k)(t),1≦k≦K be the cumulative flow on the kth pathunder scheme R. Let {tilde over (R)} be the intra-flow splittingapproach where all flows between the source and destination pair has thesame split ratio vector (p₁, . . . p_(k)) such that p_(k) isproportional to the mean traffic rate μ_(k) of X_(k)(t), i.e.:

$\begin{matrix}{p_{k} = {\frac{\mu_{k}}{{\sum\limits_{k = 1}^{K}\mu_{k}}\;}.}} & (7)\end{matrix}$

We use this split ratio vector to ensure that all links in the networkhave the same mean traffic rate under both schemes.

For a Gaussian traffic model that satisfies the superposition assumptionin Equation (3) (section 1.2.1), the intra-flow splitting approachdefined in Equation (7) has a superior performance over the flow-basedsplitting scheme R for any link in the network. Specifically, any linkwith a given bandwidth and buffer size has a smaller queue length andloss probability. In addition, for a given QoS requirement of maximumdelay and loss probability, the QoS bandwidth of the traffic on the linkis smaller.

This can be shown as follows. Under the assumption that each flow is theresult of superposition of independent and identically distributed basetraffic (section 1.2.1 and Equation (3), for a flow-based splittingscheme R, the traffic X_(k)(t) carried on path k between the source nodeS and destination node D can be written as:X _(k)(t)=μ_(k)+σ_(k) Z _(k)(t), σ_(k) ²=αμ_(k) , k=1, . . . , K

for some α>0, where Z_(k)(t) are independent stationary Gaussianprocesses identically distributed as a Gaussian process Z(t). Noticethat under intra-flow splitting approach {tilde over (R)} defined inEquation (7), the traffic carried on path k is:

${{{\overset{\sim}{X}}_{k}(t)} = {{p_{k}{\sum\limits_{k = 1}^{K}\;{X_{k}(t)}}} = {{\mu_{k}\left( {\sum\limits_{k = 1}^{K}\;\mu_{k}} \right)}^{- 1}{\sum\limits_{k = 1}^{K}\;{X_{k}(t)}}}}},$

with mean:

${{E\left( {{\overset{\sim}{X}}_{k}(t)} \right)} = {{{\mu_{k}\left( {\sum\limits_{k = 1}^{K}\;\mu_{k}} \right)}^{- 1}{\sum\limits_{k = 1}^{k}\;\mu_{k}}} = \mu_{k}}},$

and variable part identically distributed as:

${{{p_{k}\left( {\sum\limits_{i = 1}^{K}\;\sigma_{k}^{2}} \right)}^{\frac{1}{2}}{Z(t)}} = {{{p_{k}\left( {\alpha{\sum\limits_{i = 1}^{K}\;\mu_{k}}} \right)}^{\frac{1}{2}}{Z(t)}} = {p_{k}^{\frac{1}{2}}\sigma_{k}{Z(t)}}}},$

that is:

${{{\overset{\sim}{X}}_{k}(t)} = {\mu_{k} + {p_{k}^{\frac{1}{2}}\sigma_{k}{Z(t)}}}},$

so that the mean traffic rate on path k stays the same as in theflow-based splitting, but the variance is reduced by a factor of p_(k).Given the independence assumption of distinct flows in the superpositionmodel (section 1.2.1), it can be concluded that for any link in thenetwork, under the intra-flow splitting approach {tilde over (R)}, themean traffic rate stays the same but the variance is reduced with thesame correlation function. This follows from the fact that the trafficprocess μ+σ₁Z(t) always results in less queueing and hence smaller lossprobability, or less QoS bandwidth, comparing to the traffic processμ+σ₂ Z(t) with σ₂>σ₁.

Such result can be generalized to an arbitrary base traffic model.

Under the assumption that each flow is the result of superposition ofindependent and identically distributed base traffic (section 1.2.1),the intra-flow based splitting approach defined in Equation (7) has asuperior performance over the flow-based splitting scheme R for any linkin the network. Specifically, any link with a given bandwidth and buffersize has a smaller queue length and loss probability. in addition, for agiven QoS requirement of maximum delay and loss probability, the QoSbandwidth of the traffic on the link is smaller.

This can be shown as follows. Using the same notation as above, for eachpath k, 1≦k≦K between a source node S to a destination node D, letX_(k)(t), {tilde over (X)}_(k)(t) be the traffic carried under theflow-based splitting scheme R and intra-flow based splitting approach{tilde over (R)}, respectively. For simplicity, we assume that X_(k)(t),k=1, . . . , K are independent and identically distributed as B₁(t)+ . .. +B_(m)(t) for some m≧1, where B_(i)(t) is the base process defined inEquation (1). Similar arguments can be used to show for the generalcases with more complicated notations. For any link on the path k fromthe source-destination pair, we consider two scenarios: first, there isno through traffic from a different source-destination pair on the link,and second, there is through traffic from a different source-destinationpair on the link. We show that for both cases the {tilde over (R)}performs better than R.

For scenario 1, the traffic on a link on path k is X_(k)(t) under R and

$\frac{1}{k}{\sum\limits_{k = 1}^{K}\;{X_{k}(t)}}$under {tilde over (R)}. Let C be the link bandwidth and B be the buffersize. Applying the above propositions to a hypothetical two-node networkwith K parallel links with bandwidth C and buffer size B with link kcarrying traffic X_(k)(t), k=1, . . . , K, it can be seen that thequeuing performance (queue length, loss probability) for this network isworse than the same network running under a intra-flow splittingapproach with link k carrying traffic

$\frac{1}{k}{\sum\limits_{k = 1}^{K}\;{{X_{k}(t)}.}}$This is equivalent to saying that the queueing performance for a link inscenario 1 carrying traffic

${{\overset{\sim}{X}}_{k}(t)} = {\frac{1}{K}{\sum\limits_{k = 1}^{K}\;{X_{k}(t)}}}$is always better than the link carrying traffic X_(k)(t).

For scenario 2, let X_(k) ^((o)) (t), {tilde over (X)}_(k) ^((o)) (t) bethe cumulative through traffic on the link under the flow-basedsplitting scheme R and the intra-flow splitting approach {tilde over(R)}, respectively. Using similar argument as in scenario 1, we can showthat a link carrying traffic X_(k)(t)+X_(o)(t) has a worse performancecomparing to the same link carrying traffic

${{\frac{1}{K}{\sum\limits_{k = 1}^{K}\;{X_{k}(t)}}} + {X_{o}(t)}},$which by a recursive argument, implies that it has a even worseperformance then the link carrying traffic is

${\frac{1}{K}{\sum\limits_{k = 1}^{K}\;{X_{k}(t)}}} + {{{\overset{\sim}{X}}_{k}^{(o)}(t)}.}$This means intra-flowing splitting has a better performance versusflow-based splitting for links in scenario 2.

It is useful to point out differences between the results presented herefor the general network and that for the two-node network in section3.1. Here we have shown that queueing and bandwidth requirement forindividual links in a general network under the intra-flow splittingapproach are less than those under the flow-based splitting scheme,assuming the traffic superposition model in Equation 1. But in section3.1, we have shown that the total queueing and bandwidth requirement forall the links between the two nodes is minimum over all routing schemes.In addition, the splitting ratios in the split ratio vector for theintra-flow splitting approach for the network are proportional to themean path throughput, but the splitting ratios for the optimalintra-flow splitting approach for the two-node network are proportionalto the link bandwidths. In fact, the link bandwidth concept used todefine the optimal intra-flow splitting approach for the two-nodenetwork may not be easily generalizable to a general network.

That said, it should be understand that, as in the case for the two-nodenetwork, the performance gain of the intra-flow splitting overflow-based splitting scheme for a general network is also due to themultiplexing gain. This can be easily understood for the Gaussiantraffic model, where the multiplexing gain resulted from the split ratiovector defined in Equation (7) produces a traffic stream with the samemean rate but a smaller variance, which in turn leads to a betterqueueing and bandwidth performance. In the following section, wequantify the performance gain for the Gaussian traffic model, and inparticular, the popular Fractional Brownian (FBM) traffic model, usingthe notion of effective bandwidth.

3.3 Bandwidth Savings for Gaussian and Fractional Brownian TrafficModels

In this subsection, we quantify the improvements in loss probability andbandwidth savings for intra-flow splitting. First, we introduce theconcept of effective bandwidth to analyze bandwidth requirements. Then,we show the reduction in loss probability for the Gaussian models basedon the reduction in variance in intra-flow splitting approach. Finally,we show the bandwidth savings for the long range dependent FractionalBrownian Motion and the short range dependent Gaussian models.

3.3.1 Effective Bandwidth

The notion of effective bandwidth provides a measure of the resourcerequirements of a traffic stream with certain quality-of-service (QoS)constraints. Statistical properties of the traffic stream have to beconsidered as well as system parameters (e.g., buffer size, servicediscipline) and the traffic mix. A mathematical framework for effectivebandwidth has been defined based on the general expression (see, e.g.,F. Kelly, “Notes on Effective Bandwidths,” Stochastic Networks: Theoryand Applications, Oxford University Press, pages 141-168, 1996, thedisclosure of which is incorporated by reference herein):

$\begin{matrix}{{{\alpha\left( {s,t} \right)} = {{\frac{1}{st}\log\;{E\left\lbrack {\mathbb{e}}^{{sX}{\lbrack{0,t}\rbrack}} \right\rbrack}\mspace{14mu} 0} \leq s}},{t \leq \infty}} & (8)\end{matrix}$

which depends on the space parameter s and the time parameter t. In apractical use of this expression, appropriate values of s and t may bedetermined based on the QoS requirements and system parameters.

We will use the following proposition to quantify the reduction in lossprobability of and effective bandwidth for Gaussian traffic models underthe intra-flow splitting scheme.

Let L(nC, nB) be the loss probability or overflow probability for aqueue fed by n identical, independent and stationary sources on a linkwith capacity nC and buffer size nB.

Then:

${{\lim\limits_{n->\infty}{{- \frac{1}{n}}\log\;{L\left( {{nC},{nB}} \right)}}} = {\inf\limits_{t > 0}{\sup\limits_{s}\left( {{s\left( {B + {Ct}} \right)} - {{st}\;{\alpha\left( {s,t} \right)}}} \right)}}},$

where α(s, t) is the effective bandwidth defined by Equation 8.

3.3.2 Reduction in Loss Probability

In this subsection, we show that intra-flow splitting has lower lossprobability than flow-based splitting for Gaussian models. That is, forthe same network, the logarithm of the loss probability reduces by afactor of variance reduction.

For a link l with a fixed bandwidth C and buffer size B, let X₁ and X₂be the Gaussian input process for flow-based and intra-flow splittingscheme respectively. Let:X ₁(t)=μ+σ₁ Z(t), X ₂(t)=μ+σ₂ Z(t),

where Z(t) is a stationary Gaussian process. We know that σ₂≦σ₁.

Let the loss rate for the traffic processes X₁(t) on the link be e^(−v)₁ . Then the loss rate for X₂(t) on the link will be e^(−v) ₂ where v₂is approximately

$\frac{\sigma_{1}^{2}}{\sigma_{2}^{2}}{v_{1}.}$

This can be shown as follows. Without loss of generality, we assume bothσ₁, and σ₂ are very small. Otherwise, we can always define a new Z(t) asε⁻¹Z(t) to make σ₁, σ₂ arbitrary small. Notice that

${\frac{1}{\sigma_{i}^{2}}{X_{i}(t)}},{i = 1},2$is identically distributed as the sum of

$n_{i} = \frac{1}{\sigma_{i}^{2}}$independent processes which are identically distributed as μ+Z(t). Let

$\frac{1}{\sigma_{i}^{2}}$and let l_(i) be the link with bandwidth n_(i)c and buffer size n_(i)b.The queueing behavior (and hence loss probability) for the flow X_(i) onlink l is same as that of

$\frac{1}{\sigma_{i}^{2}}{X_{i}(t)}$on link l_(i). Hence applying the above proposition as σ₁, σ₂→0:

${{- \frac{1}{n_{i}}}\log\;{\mathbb{e}}^{- v_{i}}} \approx {\inf\limits_{t > 0}{\sup\limits_{s}\left( {{s\left( {b + {ct}} \right)} - {{st}\;{\alpha\left( {s,t} \right)}}} \right)}}$

Hence σ₁ ²v₁≈σ₂ ²v₂, which proves the proposition.

Hence if flow-based splitting has a loss probability d<1 and if thevariance reduces by a factor r for intra-flow splitting, then the lossprobability for intra-flow splitting will be d^(r). In particular, for atwo-node network with K parallel links where under the flow-basedsplitting, traffic on all links are independent and identicallydistributed Gaussian processes, the variance reduces by a factor of K.Hence if the loss probability for flow-based splitting is d, then theloss probability for the intra-flow splitting will be d^(K).

3.3.3 Reduction in Bandwidth

Although it is easy to characterize how the loss probability for aGaussian traffic model μ+σZ(t) changes with σ, it is quite difficult tocharacterize how the effective bandwidth will change with σ. This isbecause there is no general formula for the choice of the space and timeparameter s, t in Equation 8 for Gaussian processes. In this section, weshow the bandwidth savings for short range dependent Gaussian processfor effective bandwidth in the large buffer asymptotic regime where thetime parameter t goes to infinity. More importantly, we also show thebandwidth savings for the long-range dependent Fractional BrownianMotion (FBM), which is frequently used to model Internet traffic.

(A) Short Range Gaussian Process

For a short-range dependent Gaussian process with mean rate u and indexof dispersion v (Equation 4), the effective bandwidth for an QoSrequirement of an overflow probability exp^(−v) for a buffer size B in alarge buffer asymptotic regime is:

$C = {\mu + {\frac{vu}{2B}.}}$

(B) Fractional Brownian Motion

Let X(t)=μ+σZ(t), where Z(t) is Fractional Brownian Motion process withHurst parameter H, then the effective bandwidth for X(t) for a buffersize B and drop rate e^(−v) is:

$\begin{matrix}{C = {\mu + {{H\left( {1 - H} \right)}^{\frac{1 - H}{H}}2^{\frac{1}{2H}}\sigma^{\frac{1}{H}}v^{\frac{1}{2H}}{b^{\frac{H - 1}{H}}.}}}} & (9)\end{matrix}$

It is easy to see that the second term in Equation 9 is the additionalbandwidth required (besides the mean throughput) for the variable partof the FBM process, σZ(t), to satisfy the given QoS criteria, and wecall this term the variable bandwidth. Notice that the variablebandwidth is proportional to

$\sigma^{\frac{1}{H}}.$Therefore, if the variance is reduced by a factor of r, then thevariable bandwidth will be reduced by a factor of

$r^{\frac{1}{2H}}.$This variable bandwidth reduction factor does not depend on the QoScriteria (loss probability e^(−v) and buffer size B). As an example, fora two node network with K parallel links where under the flow-basedsplitting, traffic on all links are independent and identicallydistributed FBM processes, the index of dispersion v reduces by a factorK for intra-flow splitting. Hence the variable bandwidth requiredreduces by a factor

$K^{\frac{1}{2H}}.$In case of H=1, i.e, the Brownian motion, there is a saving of (K−1)/Kin variable bandwidth for intra-flow splitting compared to flow basedsplitting.4. Illustrative Node Implementation

FIG. 4 illustrates an implementation of a node, according to anembodiment of the invention. More particularly, FIG. 4 illustrates node400, which may act as a source node, an intermediate node, and/or adestination node in a network (e.g., FIG. 1B) in which intra-flowsplitting principles of the invention are employed. It is to beappreciated that one or more of the routing methodologies of theembodiments described herein may be implemented via the computing systemof network node 400. For example, the methodology of FIG. 2 may beimplemented in network node 400. Other types of node configurations maybe used, as will be appreciated by those skilled in the art, and a givennetwork may include many nodes with differing configurations.

Generally, as shown, node 400 is configured so as to include processor410 coupled to memory 420. Processor 410 may comprise a microprocessor,a microcontroller, a central processing unit (CPU), anapplication-specific integrated circuit (ASIC) or other type ofprocessing device, as well as portions or combinations of such devices.Memory 420 may comprise an electronic random access memory (RAM), aread-only memory (ROM) or other type of storage device, as well asportions or combinations of such devices. The memory may be used tostore software that is executed by or otherwise utilized by theprocessor in implementing at least a portion of a routing methodology inaccordance with the present embodiments.

Node 400 may be viewed as an example of a “processing device.” Such aprocessing device may be implemented in the form of one or moreintegrated circuits, as well as in the form of other types of hardware,software or firmware, in any combination.

It is to be appreciated that node 400 is considerably simplified forpurposes of illustration, and may include other elements, not explicitlyshown. For example, node 400 may include conventional interfaces and/orprotocols for transmitting data to, and receiving data from, one or moreother nodes in the network.

Although illustrative embodiments of the present invention have beendescribed herein with reference to the accompanying drawings, it is tobe understood that the invention is not limited to those preciseembodiments, and that various other changes and modifications may bemade by one skilled in the art without departing from the scope orspirit of the invention.

1. A method of processing traffic flows at a node in a network,comprising the steps of: obtaining by the node a plurality of trafficflows, each of the traffic flows comprising multiple packets or bytes;splitting by the node each of the plurality of traffic flows into atleast two sub-flows, wherein each of the at least two sub-flowscomprises a portion of the multiple packets or bytes from its respectivetraffic flow; and routing by the node the packets or bytes of the atleast two sub-flows respectively on at least two paths in the network;wherein the routing step further comprises combining at least one of thesub-flows of each of at least two of the plurality of traffic flows androuting the combined packets or bytes on one of the paths; wherein eachtraffic flow between the node and a given destination node is splitbased on a given split ratio vector into a plurality of sub-flowscorresponding to respective ones of a plurality of paths between thenode and the given destination node, and further wherein the given splitratio vector is configured such that the portion of the multiple packetsor bytes assigned to each of the plurality of sub-flows is based atleast in part on a ratio between a mean traffic rate of a cumulativeflow on the respective corresponding path and a sum of mean trafficrates of cumulative flows on the plurality of paths.
 2. The method ofclaim 1, wherein the routing step further comprises combining others ofthe sub-flows of at least a portion of the plurality of traffic flowsand routing the combined packets or bytes on another of the paths. 3.The method of claim 1, wherein the splitting of the plurality of flowsinto at least two sub-flows and routing the packets or bytes of the atleast two sub-flows respectively on at least two paths in the networkreduces a variance associated with at least one of the traffic flows. 4.The method of claim 3, wherein the splitting of the flows into at leasttwo sub-flows and routing the packets or bytes of the at least twosub-flows respectively on at least two paths in the network reduces aloss probability associated with at least one of the traffic flows. 5.The method of claim 4, wherein the loss probability is d, where d is theloss probability of flow-based splitting and r is the reduction invariance for intra-flow splitting.
 6. The method of claim 4, wherein thereduction is realizable at a network design time or a provisioning time.7. The method of claim 2, wherein the splitting of the traffic flowsinto sub-flows, combining ones of the sub-flows, combining others of thesub-flows, and routing the packets or bytes of the combined sub-flows onat least two paths in the network reduces a bandwidth requirementassociated with the traffic flows.
 8. The method of claim 7, wherein avariable bandwidth is reduced by a factor of $r^{\frac{1}{2H}},$ where ris a variance reduction factor and H is a Hurst parameter.
 9. The methodof claim 7, wherein the reduction is realizable at a network design timeor a provisioning time.
 10. The method of claim 1, wherein at least oneof the traffic flows comprises a long range dependent traffic flow or ashort range dependent traffic flow.
 11. Apparatus for processing trafficflows at a node of a network, comprising: a memory; and a processorcoupled to the memory and operative to: (i) obtain a plurality oftraffic flows, each of the plurality of traffic flows comprisingmultiple packets or bytes; (ii) split each of the plurality of trafficflows into at least two sub-flows based on a split ratio vector, whereineach of the at least two sub-flows comprises a portion of the multiplepackets or bytes from its respective traffic flow; and (iii) route thepackets or bytes of the at least two sub-flows respectively on at leasttwo paths in the network; wherein the routing operation furthercomprises combining at least one of the sub-flows of each of at leasttwo of the plurality of traffic flows and routing the combined packetsor bytes on one of the paths; wherein each traffic flow between the nodeand a given destination node is split based on a given split ratiovector into a plurality of sub-flows corresponding to respective ones ofa plurality of paths between the node and the given destination node,and further wherein the given split ratio vector is configured such thatthe portion of the multiple packets or bytes assigned to each of theplurality of sub-flows is based at least in part on a ratio between amean traffic rate of a cumulative flow on the respective correspondingpath and a sum of mean traffic rates of cumulative flows on theplurality of paths.
 12. The apparatus of claim 11, wherein the routingoperation further comprises combining others of the sub-flows of atleast a portion of the plurality of traffic flows and routing thecombined packets or bytes on another of the paths.
 13. The apparatus ofclaim 11, wherein the splitting of the plurality of flows into at leasttwo sub-flows and routing the packets or bytes of the at least twosub-flows respectively on at least two paths in the network reduces avariance associated with at least one of the traffic flows.
 14. Theapparatus of claim 12, wherein the splitting of the traffic flows intosub-flows, combining ones of the sub-flows, combining others of thesub-flows, and routing the packets or bytes of the combined sub-flows onat least two paths in the network reduces a bandwidth requirementassociated with the traffic flows.
 15. The apparatus of claim 11,wherein at least one of the traffic flows comprises a long rangedependent traffic flow or a short range dependent traffic flow.
 16. Amulti-path routing-capable node of a mesh-type network, comprising: amemory; and a processor coupled to the memory and operative to: (i)obtain a plurality of traffic flows, each of the traffic flowscomprising multiple packets or bytes; (ii) split each of the pluralityof flows into at least two sub-flows based on a split ratio vector,wherein each of the sub-flows comprises a portion of the multiplepackets or bytes from its respective traffic flow; and (iii) combiningones of the sub-flows of each of the plurality of traffic flows androuting the combined packets or bytes on one path of the mesh-typenetwork, and combining others of the sub-flows of each of the at leasttwo traffic flows and routing the combined packets or bytes on anotherpath of the mesh-type network, such that at least one of the sub-flowsof each of at least two of the plurality of traffic flows are combinedand routed on a given path of the mesh-type network; wherein eachtraffic flow between the node and a given destination node is splitbased on a given split ratio vector into a plurality of sub-flowscorresponding to respective ones of a plurality of paths between thenode and the given destination node, and further wherein the given splitratio vector is configured such that the portion of the multiple packetsor bytes assigned to each of the plurality of sub-flows is based atleast in part on a ratio between a mean traffic rate of a cumulativeflow on the respective corresponding path and a sum of mean trafficrates of cumulative flows on the plurality of paths.